Skip to content

ci: adopt benchmark-qualified Koru judge - #4

Open
tom-sapletta-com wants to merge 7 commits into
mainfrom
ticket-018-koru-review
Open

ci: adopt benchmark-qualified Koru judge#4
tom-sapletta-com wants to merge 7 commits into
mainfrom
ticket-018-koru-review

Conversation

@tom-sapletta-com

@tom-sapletta-com tom-sapletta-com commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • switch the Koru semantic judge from stale DeepSeek to benchmark-qualified google/gemini-3.1-pro-preview
  • keep Vallm semantic/security review fail-closed while leaving TypeScript regression to the required verify job
  • record commit-bound no-source and live two-file review evidence

Local validation

  • npm run verify:workflows — PASS
  • npm run verify — PASS (335 tests, 334 passed, 1 JDK-dependent skip)
  • focused model tests — PASS 2/2
  • Vallm compatibility probe — deterministic/security PASS 2/2 after lowercase language normalization
  • make governance — expected inherited ticket-019 findings only; no ticket-018 scope/secret finding

Timeout and failure policy

  • complete GitHub job: 10 minutes
  • Vallm/LiteLLM review: 420 seconds; response cap: 8192 tokens
  • provider retries forced to zero, so HTTP errors including 404 fail immediately
  • Vallm 0.1.94 TypeScript language ID normalized before tree-sitter parsing
  • semantic info/warnings are advisory only with a passing file verdict; semantic errors and all deterministic, provider, malformed/missing-result and timeout findings remain blocking and visible in JSON

Live evidence

Repository-scoped OPENROUTER_API_KEY was rotated through gh stdin after explicit user authorization. Dispatch 30714664770 reached openrouter/google/gemini-3.1-pro-preview for both PR #3 TypeScript files without authentication or provider-limit errors. Both file-level verdicts were pass; that run exposed the parser and aggregate-policy defects repaired by this PR.

Repaired live evidence

Dispatch 30746421293 reviewed PR #3 range 2e87205..6b79527 in 1m24s. The attested report records Gemini, both expected TypeScript files, verdict pass, exit 0, 2/2 passed, zero blocking/parser/provider findings, and five retained advisory semantic observations. Vallm original exit 2 remains recorded before policy normalization.

Remaining blockers

  • inherited ticket-019 governance conflicts
  • independent protected review/merge evidence

Follow-up failure-path validation

  • actual LiteLLM HTTP stack + local 404 endpoint: in ~705 ms, exactly one request,
  • slow local endpoint + 0.5 s request limit: in ~799 ms, exactly one request
  • fresh

todo2code@0.5.1 verify
npm run check && npm run verify:no-llm && npm run verify:modules && npm run verify:env && npm run verify:workflows && npm run verify:generated-analysis && npm run verify:structured-responses && npm run build && npm run verify:schemas && npm test

todo2code@0.5.1 check
tsc -p tsconfig.json --noEmit

todo2code@0.5.1 verify:no-llm
node scripts/verify-no-llm-imports.mjs

LLM boundary verified transitively from 9 deterministic entrypoints across 38 modules.

todo2code@0.5.1 verify:modules
node scripts/verify-module-boundaries.mjs

Module boundaries verified: 114 modules, 521 internal imports, no cycles, core is independent.

todo2code@0.5.1 verify:env
node scripts/verify-env-contract.mjs

Environment contract verified: 75 code/Docker variables, 75 documented keys, no duplicates.

todo2code@0.5.1 verify:workflows
node scripts/verify-workflow-yaml.mjs

Workflow YAML verified: 2 file(s), no duplicate top-level keys.

todo2code@0.5.1 verify:generated-analysis
node scripts/verify-generated-analysis.mjs

{"filesChecked":19,"untrackedInputsChecked":0,"status":"ok"}

todo2code@0.5.1 verify:structured-responses
node scripts/verify-structured-responses.mjs

{"structuredCalls":7,"rawCalls":0,"status":"ok"}

todo2code@0.5.1 build
tsc -p tsconfig.json

todo2code@0.5.1 verify:schemas
node scripts/generate-response-schemas.mjs --check

{"schema":"schemas/document-extraction-response.schema.json","status":"ok"}

todo2code@0.5.1 test
node --test --test-concurrency=4 dist/test/*.test.js

TAP version 13

[t2c:a2a] listening on 127.0.0.1:38195

[t2c:a2a] listening on 127.0.0.1:37603

Subtest: A2A governed-intake skill completes accepted commands and rejects domain failures

ok 1 - A2A governed-intake skill completes accepted commands and rejects domain failures

duration_ms: 133.049905
...

[t2c:a2a] listening on 127.0.0.1:35497

Subtest: A2A v1.0 card, versioning, task methods and cursor pagination are coherent

ok 2 - A2A v1.0 card, versioning, task methods and cursor pagination are coherent

duration_ms: 140.624917
...

[t2c:a2a] listening on 127.0.0.1:41177

Subtest: A2A bearer authentication is declared with v1 security objects and enforced

ok 3 - A2A bearer authentication is declared with v1 security objects and enforced

duration_ms: 32.579244
...

[t2c:a2a] listening on 127.0.0.1:41259

[t2c:a2a] listening on 127.0.0.1:32825

[t2c:a2a] listening on 127.0.0.1:43233

Subtest: A2A file task store survives restart and preserves idempotency across replicas

ok 4 - A2A file task store survives restart and preserves idempotency across replicas

duration_ms: 110.351699
...

Subtest: the A2A bind default is loopback and a wildcard bind is not mistaken for it

ok 5 - the A2A bind default is loopback and a wildcard bind is not mistaken for it

duration_ms: 1.834447
...

Subtest: Go adapter records package, imports, types, functions and methods

ok 6 - Go adapter records package, imports, types, functions and methods

duration_ms: 191.156694
...

Subtest: Go facts are deterministic observations, not inferences

ok 7 - Go facts are deterministic observations, not inferences

duration_ms: 144.172903
...

Subtest: Go adapter marks exported symbols and reports calls in scope

ok 8 - Go adapter marks exported symbols and reports calls in scope

duration_ms: 113.207635
...

Subtest: Go extraction is skipped without cost when a tree holds no Go sources

ok 9 - Go extraction is skipped without cost when a tree holds no Go sources

duration_ms: 14.56097
...

Subtest: A missing Go toolchain degrades to a warning instead of failing the run

ok 10 - A missing Go toolchain degrades to a warning instead of failing the run

duration_ms: 11.266279
...

Subtest: Rust adapter records uses, types, functions, methods, values and calls

ok 11 - Rust adapter records uses, types, functions, methods, values and calls

duration_ms: 104.750893
...

Subtest: Java adapter records packages, imports, types, fields, methods and calls

ok 12 - Java adapter records packages, imports, types, fields, methods and calls # SKIP JDK not installed

duration_ms: 8.966518
...

Subtest: Java and Rust adapters skip toolchain startup when no matching sources exist

ok 13 - Java and Rust adapters skip toolchain startup when no matching sources exist

duration_ms: 21.042575
...

Subtest: Missing Java and Rust toolchains degrade to explicit warnings

ok 14 - Missing Java and Rust toolchains degrade to explicit warnings

duration_ms: 51.660978
...

Subtest: PHP syntax adapter records namespaces, imports, types, functions, methods and calls

ok 15 - PHP syntax adapter records namespaces, imports, types, functions, methods and calls

duration_ms: 101.635199
...

Subtest: PHP adapter skips runtime startup when no PHP source exists

ok 16 - PHP adapter skips runtime startup when no PHP source exists

duration_ms: 16.947268
...

Subtest: Missing PHP runtime degrades to an explicit warning

ok 17 - Missing PHP runtime degrades to an explicit warning

duration_ms: 9.744386
...

Subtest: Invalid PHP syntax is reported without aborting extraction

ok 18 - Invalid PHP syntax is reported without aborting extraction

duration_ms: 75.710835
...

Subtest: AST extractor reads TypeScript and Python facts

ok 19 - AST extractor reads TypeScript and Python facts

duration_ms: 110.625447
...

Subtest: CLI command help is successful and non-mutating

ok 20 - CLI command help is successful and non-mutating

duration_ms: 2375.182123
...

Subtest: TypeScript and Python CLIs execute the same intake command/query handler

ok 21 - TypeScript and Python CLIs execute the same intake command/query handler

duration_ms: 1629.798729
...

Subtest: CLI summarize exposes deterministic, prefer-llm and require-llm modes

ok 22 - CLI summarize exposes deterministic, prefer-llm and require-llm modes

duration_ms: 4100.87357
...

Subtest: CLI propose-todo, render-todo and apply-todo return JSON and preserve a no-op TODO

ok 23 - CLI propose-todo, render-todo and apply-todo return JSON and preserve a no-op TODO

duration_ms: 3147.502079
...

Subtest: CLI watch reads TASK.md by default, disables summary LLM and reacts to a live file change

ok 24 - CLI watch reads TASK.md by default, disables summary LLM and reacts to a live file change

duration_ms: 2086.17955
...

Subtest: proposeCodeChangePlans materialises grounded plans from PLANNED_NOT_IMPLEMENTED

ok 25 - proposeCodeChangePlans materialises grounded plans from PLANNED_NOT_IMPLEMENTED

duration_ms: 31.288063
...

Subtest: code-change title preserves the leading action of a compound intent

ok 26 - code-change title preserves the leading action of a compound intent

duration_ms: 2.31415
...

Subtest: proposeCodeChangePlans is deterministic for the same evidence

ok 27 - proposeCodeChangePlans is deterministic for the same evidence

duration_ms: 3.980339
...

Subtest: a plan creates a missing file and modifies an existing one

ok 28 - a plan creates a missing file and modifies an existing one

duration_ms: 11.564583
...

Subtest: a plan never invents a repository-root file from a bare filename

ok 29 - a plan never invents a repository-root file from a bare filename

duration_ms: 3.953313
...

Subtest: a plan whose every path is bare and missing is withheld entirely

ok 30 - a plan whose every path is bare and missing is withheld entirely

duration_ms: 1.724869
...

Subtest: a plan never targets a location outside the repository tree

ok 31 - a plan never targets a location outside the repository tree

duration_ms: 0.312334
...

Subtest: the repository probe never proposes creating a file outside the root

ok 32 - the repository probe never proposes creating a file outside the root

duration_ms: 1.263122
...

Subtest: bounded plan sets prefer explicit TODO work over historical changelog audit

ok 33 - bounded plan sets prefer explicit TODO work over historical changelog audit

duration_ms: 3.550929
...

Subtest: evaluateCodeChangeAcceptance passes when targeted diagnostics clear

ok 34 - evaluateCodeChangeAcceptance passes when targeted diagnostics clear

duration_ms: 5.921532
...

Subtest: evaluateCodeChangeAcceptance fails while the plan is still open

ok 35 - evaluateCodeChangeAcceptance fails while the plan is still open

duration_ms: 1.819317
...

Subtest: Plans without repository paths are not invented

ok 36 - Plans without repository paths are not invented

duration_ms: 0.986083
...

Subtest: Non-repository paths are ignored instead of aborting code-change planning

ok 37 - Non-repository paths are ignored instead of aborting code-change planning

duration_ms: 1.280131
...

Subtest: Acceptance rejects ungrounded paths, missing provenance and inconsistent verdicts

ok 38 - Acceptance rejects ungrounded paths, missing provenance and inconsistent verdicts

duration_ms: 4.742995
...

Subtest: createCodeChangeSourcePatch is deterministic and path-bound

ok 39 - createCodeChangeSourcePatch is deterministic and path-bound

duration_ms: 4.745655
...

Subtest: applyUnifiedDiffToText creates and modifies files from hunks

ok 40 - applyUnifiedDiffToText creates and modifies files from hunks

duration_ms: 0.728403
...

Subtest: applyCodeChangeSourcePatch requires approval and is idempotent

ok 41 - applyCodeChangeSourcePatch requires approval and is idempotent

duration_ms: 14.276191
...

Subtest: applyCodeChangeSourcePatch preflights diffs and refuses symlink escapes

ok 42 - applyCodeChangeSourcePatch preflights diffs and refuses symlink escapes

duration_ms: 5.517787
...

Subtest: createCodeChangeSourcePatchSet covers every plan

ok 43 - createCodeChangeSourcePatchSet covers every plan

duration_ms: 2.901765
...

Subtest: createCodeChangeReviewPatch is hash-stable and lists grounded paths

ok 44 - createCodeChangeReviewPatch is hash-stable and lists grounded paths

duration_ms: 2.789541
...

Subtest: CLI proposes and evaluates a grounded code-change plan through persisted JSON

ok 45 - CLI proposes and evaluates a grounded code-change plan through persisted JSON

duration_ms: 3226.923974
...

Subtest: isUsefulCodeChangePath rejects vendored, binary and analysis dump paths

ok 46 - isUsefulCodeChangePath rejects vendored, binary and analysis dump paths

duration_ms: 0.475118
...

Subtest: proposeCodeChangePlans skips diagnostics that only name junk paths

ok 47 - proposeCodeChangePlans skips diagnostics that only name junk paths

duration_ms: 0.925965
...

Subtest: Published code-change JSON schemas require provenance, risk and rollback

ok 48 - Published code-change JSON schemas require provenance, risk and rollback

duration_ms: 1.832967
...

Subtest: participant registry maps stable IDs to Git/A2A identifiers without display-name guessing

ok 49 - participant registry maps stable IDs to Git/A2A identifiers without display-name guessing

duration_ms: 52.365262
...

Subtest: participant registry rejects ambiguous external identifiers

ok 50 - participant registry rejects ambiguous external identifiers

duration_ms: 0.672329
...

Subtest: trusted intake persists roles across tickets and fails closed without rejected writes

ok 51 - trusted intake persists roles across tickets and fails closed without rejected writes

duration_ms: 117.61495
...

Subtest: agents cannot create human projections and a damaged event hash fails verification

ok 52 - agents cannot create human projections and a damaged event hash fails verification

duration_ms: 25.243968
...

Subtest: TypeScript and dependency-free Python codecs share golden bytes and preserve unknown fields

ok 53 - TypeScript and dependency-free Python codecs share golden bytes and preserve unknown fields

duration_ms: 80.695325
...

Subtest: communication enrichment preserves runtime identity, source, ticket and epistemic class

ok 54 - communication enrichment preserves runtime identity, source, ticket and epistemic class

duration_ms: 66.589084
...

Subtest: communication enrichment corrects one rejected structured response without weakening validation

ok 55 - communication enrichment corrects one rejected structured response without weakening validation

duration_ms: 15.570318
...

Subtest: communication prefer-llm fallback is explicit and require-llm rejects

ok 56 - communication prefer-llm fallback is explicit and require-llm rejects

duration_ms: 20.178406
...

Subtest: project/ communication is attributed per human and agent and checked against Git evidence

ok 57 - project/ communication is attributed per human and agent and checked against Git evidence

duration_ms: 178.309318
...

Subtest: governance user-* and ai-* files become typed participant intent without ingesting ticket evidence

ok 58 - governance user-* and ai-* files become typed participant intent without ingesting ticket evidence

duration_ms: 27.681578
...

Subtest: unstructured governance participant content is rejected with an owner-specific migration warning

ok 59 - unstructured governance participant content is rejected with an owner-specific migration warning

duration_ms: 4.560874
...

Subtest: opposite wording about different explicit files is not treated as an intent conflict

ok 60 - opposite wording about different explicit files is not treated as an intent conflict

duration_ms: 7.967587
...

Subtest: missing response owners use explicit role sentinels without inventing participants

ok 61 - missing response owners use explicit role sentinels without inventing participants

duration_ms: 11.718646
...

Subtest: communication extractor reports unresolved identity instead of inventing an actor

ok 62 - communication extractor reports unresolved identity instead of inventing an actor

duration_ms: 5.699314
...

Subtest: communication extractor ignores generic generated analysis under project/

ok 63 - communication extractor ignores generic generated analysis under project/

duration_ms: 11.552685
...

Subtest: configuration converter covers JSON, TOML, Docker and CI workflow declarations

ok 64 - configuration converter covers JSON, TOML, Docker and CI workflow declarations

duration_ms: 40.551884
...

Subtest: configuration converter emits a deterministic file aggregate for an empty configuration

ok 65 - configuration converter emits a deterministic file aggregate for an empty configuration

duration_ms: 5.560812
...

Subtest: splitLines treats a trailing newline as a terminator, not an extra line

ok 66 - splitLines treats a trailing newline as a terminator, not an extra line

duration_ms: 1.689699
...

Subtest: Identical inputs produce no hunks

ok 67 - Identical inputs produce no hunks

duration_ms: 0.596966
...

Subtest: A modified line keeps both sides addressable by original line number

ok 68 - A modified line keeps both sides addressable by original line number

duration_ms: 0.403045
...

Subtest: Pure insertion and pure deletion are not reported as replacements

ok 69 - Pure insertion and pure deletion are not reported as replacements

duration_ms: 0.309081
...

Subtest: Empty-to-content and content-to-empty are handled as block changes

ok 70 - Empty-to-content and content-to-empty are handled as block changes

duration_ms: 0.306701
...

Subtest: Context width controls hunk size

ok 71 - Context width controls hunk size

duration_ms: 0.433603
...

Subtest: Nearby changes merge into a single hunk

ok 72 - Nearby changes merge into a single hunk

duration_ms: 0.329865
...

Subtest: Distant changes stay in separate hunks

ok 73 - Distant changes stay in separate hunks

duration_ms: 0.338893
...

Subtest: Oversized inputs fall back to a bounded block replace

ok 74 - Oversized inputs fall back to a bounded block replace

duration_ms: 0.617573
...

Subtest: Unified output carries a well formed hunk header

ok 75 - Unified output carries a well formed hunk header

duration_ms: 0.497057
...

Subtest: Side-by-side rows pair deletions with insertions

ok 76 - Side-by-side rows pair deletions with insertions

duration_ms: 0.328081
...

Subtest: Unbalanced change runs leave one side empty rather than misaligning

ok 77 - Unbalanced change runs leave one side empty rather than misaligning

duration_ms: 1.111211
...

Subtest: Renderers escape source markup

ok 78 - Renderers escape source markup

duration_ms: 1.490097
...

Subtest: SVG rendering caps rows and reports the remainder

ok 79 - SVG rendering caps rows and reports the remainder

duration_ms: 2.02803
...

Subtest: Reality view keys topics by target and records lane presence

ok 80 - Reality view keys topics by target and records lane presence

duration_ms: 20.325596
...

Subtest: A topic holding declared and observed records is never reported as planned-only

ok 81 - A topic holding declared and observed records is never reported as planned-only

duration_ms: 3.784482
...

Subtest: Reality coverage stays open when a shared path has unrelated capabilities

ok 82 - Reality coverage stays open when a shared path has unrelated capabilities

duration_ms: 2.264223
...

Subtest: Shared-path relations do not collapse unrelated files into one topic

ok 83 - Shared-path relations do not collapse unrelated files into one topic

duration_ms: 3.85859
...

Subtest: Reality view is deterministic for identical input

ok 84 - Reality view is deterministic for identical input

duration_ms: 3.073571
...

Subtest: Reality SVG escapes topic labels

ok 85 - Reality SVG escapes topic labels

duration_ms: 1.794578
...

Subtest: graph diff detects changed source identities, additions and SVG-safe labels

ok 86 - graph diff detects changed source identities, additions and SVG-safe labels

duration_ms: 27.5652
...

Subtest: graph diff is empty for graphs with identical evidence

ok 87 - graph diff is empty for graphs with identical evidence

duration_ms: 2.203028
...

Subtest: file diff emits deterministic unified, SVG and HTML views

ok 88 - file diff emits deterministic unified, SVG and HTML views

duration_ms: 2.315685
...

Subtest: intent-vs-reality builds an explainable SVG and Markdown projection

ok 89 - intent-vs-reality builds an explainable SVG and Markdown projection

duration_ms: 5.745638
...

Subtest: a targetless declaration is filed under the single module it links to

ok 90 - a targetless declaration is filed under the single module it links to

duration_ms: 3.413625
...

Subtest: a declaration touching several modules keeps its own topic

ok 91 - a declaration touching several modules keeps its own topic

duration_ms: 3.621179
...

Subtest: a topic about a document does not absorb the statements written in it

ok 92 - a topic about a document does not absorb the statements written in it

duration_ms: 2.96805
...

Subtest: semantically aligned configuration topics retain their evidence grade

ok 93 - semantically aligned configuration topics retain their evidence grade

duration_ms: 2.5496
...

Subtest: A record claiming line 1 is re-anchored to the line carrying its statement

ok 94 - A record claiming line 1 is re-anchored to the line carrying its statement

duration_ms: 52.353844
...

Subtest: An already correct line is kept and not reported as re-anchored

ok 95 - An already correct line is kept and not reported as re-anchored

duration_ms: 8.094266
...

Subtest: An empty target is backfilled from the statement text

ok 96 - An empty target is backfilled from the statement text

duration_ms: 12.889046
...

Subtest: A target supplied by the model is never overwritten

ok 97 - A target supplied by the model is never overwritten

duration_ms: 9.974896
...

Subtest: An unclassified action and modality are derived from the statement

ok 98 - An unclassified action and modality are derived from the statement

duration_ms: 6.824687
...

Subtest: A classified action from the model wins over the heuristic

ok 99 - A classified action from the model wins over the heuristic

duration_ms: 12.18531
...

Subtest: An action that stays unclassifiable is reported as a missing field

ok 100 - An action that stays unclassifiable is reported as a missing field

duration_ms: 7.441539
...

Subtest: A placeholder object is treated as a gap, not as content

ok 101 - A placeholder object is treated as a gap, not as content

duration_ms: 10.724431
...

Subtest: Every repair is attributable through epistemic.basis

ok 102 - Every repair is attributable through epistemic.basis

duration_ms: 11.498534
...

Subtest: deterministic documentation baseline records headings, code blocks and explicit references

ok 103 - deterministic documentation baseline records headings, code blocks and explicit references

duration_ms: 23.09708
...

Subtest: deterministic documentation preserves Polish prohibition polarity

ok 104 - deterministic documentation preserves Polish prohibition polarity

duration_ms: 9.762369
...

Subtest: deterministic documentation resolves a unique bare filename against the repository

ok 105 - deterministic documentation resolves a unique bare filename against the repository

duration_ms: 4.88998
...

Subtest: documentation prose resolves a bare filename to its repository location

ok 106 - documentation prose resolves a bare filename to its repository location

duration_ms: 4.749655
...

Subtest: a nested checkout does not shadow the repository copy of a documented file

ok 107 - a nested checkout does not shadow the repository copy of a documented file

duration_ms: 8.796265
...

Subtest: AST cache is incremental by path and source content hash

ok 108 - AST cache is incremental by path and source content hash

duration_ms: 36.346025
...

Subtest: AST cache rejects corrupt entries and recomputes authoritative records

ok 109 - AST cache rejects corrupt entries and recomputes authoritative records

duration_ms: 8.407583
...

Subtest: AST cache can be bypassed without changing extraction output

ok 110 - AST cache can be bypassed without changing extraction output

duration_ms: 5.489153
...

Subtest: successful external AST adapter is skipped on a warm manifest hit

ok 111 - successful external AST adapter is skipped on a warm manifest hit

duration_ms: 68.187217
...

Subtest: documentation chunks cache independently while provider calls remain live

ok 112 - documentation chunks cache independently while provider calls remain live

duration_ms: 35.314472
...

Subtest: generated analysis replaces its source root with a stable token

ok 113 - generated analysis replaces its source root with a stable token

duration_ms: 69.09302
...

Subtest: generated analysis root normalization refuses the filesystem root

ok 114 - generated analysis root normalization refuses the filesystem root

duration_ms: 36.309173
...

Subtest: generated analysis rejects references to untracked input

ok 115 - generated analysis rejects references to untracked input

duration_ms: 76.356725
...

Subtest: generated analysis accepts outputs independent of untracked input

ok 116 - generated analysis accepts outputs independent of untracked input

duration_ms: 90.265995
...

Subtest: generated analysis accepts an untracked filename already quoted by tracked evidence

ok 117 - generated analysis accepts an untracked filename already quoted by tracked evidence

duration_ms: 108.966923
...

Subtest: generated analysis rejects temporary paths and unavailable validators

ok 118 - generated analysis rejects temporary paths and unavailable validators

duration_ms: 82.013733
...

Subtest: generated README metadata is synchronized from package.json and stays idempotent

ok 119 - generated README metadata is synchronized from package.json and stays idempotent

duration_ms: 109.305934
...

Subtest: generated README synchronization fails closed when the template drifts

ok 120 - generated README synchronization fails closed when the template drifts

duration_ms: 36.16117
...

Subtest: generated README synchronization rejects output outside the project root

ok 121 - generated README synchronization rejects output outside the project root

duration_ms: 32.778528
...

Subtest: Git extractor emits one record per requested commit

ok 122 - Git extractor emits one record per requested commit

duration_ms: 277.011863
...

Subtest: An empty repository degrades to a warning instead of failing the run

ok 123 - An empty repository degrades to a warning instead of failing the run

duration_ms: 12.066202
...

Subtest: versioned gold dataset reports perfect offline quality and repeated-run stability

ok 124 - versioned gold dataset reports perfect offline quality and repeated-run stability

duration_ms: 177.097726
...

Subtest: gold linking reports exact-target and capability-topic quality separately

ok 125 - gold linking reports exact-target and capability-topic quality separately

duration_ms: 82.710325
...

Subtest: gold capability-topic support is large enough to detect a floor regression

ok 126 - gold capability-topic support is large enough to detect a floor regression

duration_ms: 67.985971
...

Subtest: gold known gaps are measured and kept out of precision and recall

ok 127 - gold known gaps are measured and kept out of precision and recall

duration_ms: 70.063407
...

Subtest: gold reports cross-language positives and hard negatives as a separate cohort

ok 128 - gold reports cross-language positives and hard negatives as a separate cohort

duration_ms: 73.882323
...

Subtest: gold diagnostics separate a false DONE claim from an evidenced one

ok 129 - gold diagnostics separate a false DONE claim from an evidenced one

duration_ms: 61.394025
...

Subtest: gold v1 stays evaluable after the v2 contract extension

ok 130 - gold v1 stays evaluable after the v2 contract extension

duration_ms: 27.331005
...

Subtest: gold loader rejects unsupported dataset versions

ok 131 - gold loader rejects unsupported dataset versions

duration_ms: 0.708722
...

Subtest: gold evaluator rejects unknown linking cohorts

ok 132 - gold evaluator rejects unknown linking cohorts

duration_ms: 6.01285
...

Subtest: gold v2 must declare diagnostics coverage

ok 133 - gold v2 must declare diagnostics coverage

duration_ms: 1.503149
...

Subtest: published gold schema matches the runtime contract

ok 134 - published gold schema matches the runtime contract

duration_ms: 1.821159
...

Subtest: gold evaluator rejects fixture files outside its temporary workspace

ok 135 - gold evaluator rejects fixture files outside its temporary workspace

duration_ms: 8.414502
...

Subtest: Linker connects plan, Git claim and AST fact

ok 136 - Linker connects plan, Git claim and AST fact

duration_ms: 16.304066
...

Subtest: Linker connects prose intent to a module through three grounded capability topics

ok 137 - Linker connects prose intent to a module through three grounded capability topics

duration_ms: 1.123335
...

Subtest: Linker does not connect a module on one generic topic alone

ok 138 - Linker does not connect a module on one generic topic alone

duration_ms: 1.010555
...

Subtest: An existing target path does not prove an unrelated capability

ok 139 - An existing target path does not prove an unrelated capability

duration_ms: 1.744485
...

Subtest: An existing target path plus an AST capability proves implementation

ok 140 - An existing target path plus an AST capability proves implementation

duration_ms: 1.658577
...

Subtest: Diagnostics distinguish descriptive documentation from prescriptive requirements

ok 141 - Diagnostics distinguish descriptive documentation from prescriptive requirements

duration_ms: 1.801252
...

Subtest: A changelog entry naming an extracted documentation file has release evidence

ok 142 - A changelog entry naming an extracted documentation file has release evidence

duration_ms: 1.360635
...

Subtest: Diagnostics ignore non-actionable changelog mechanics but retain release claims

ok 143 - Diagnostics ignore non-actionable changelog mechanics but retain release claims

duration_ms: 6.242388
...

Subtest: Grounded conclusion and TODO proposal contracts accept traceable values

ok 144 - Grounded conclusion and TODO proposal contracts accept traceable values

duration_ms: 7.883767
...

Subtest: Stable IDs ignore ordering noise but change with semantic content

ok 145 - Stable IDs ignore ordering noise but change with semantic content

duration_ms: 1.389677
...

Subtest: Validators reject ungrounded citations and stale semantic IDs

ok 146 - Validators reject ungrounded citations and stale semantic IDs

duration_ms: 2.044739
...

Subtest: Generation metadata exposes LLM failures instead of silently masking them

ok 147 - Generation metadata exposes LLM failures instead of silently masking them

duration_ms: 1.212347
...

Subtest: TODO proposal collections enforce dependency integrity

ok 148 - TODO proposal collections enforce dependency integrity

duration_ms: 1.255271
...

Subtest: Published JSON schemas identify all grounded output contract versions

ok 149 - Published JSON schemas identify all grounded output contract versions

duration_ms: 7.596157
...

Subtest: Blank lines and comments produce no rules

ok 150 - Blank lines and comments produce no rules

duration_ms: 1.716916
...

Subtest: A pattern without a slash matches at any depth

ok 151 - A pattern without a slash matches at any depth

duration_ms: 0.471175
...

Subtest: A leading slash anchors the pattern to the root

ok 152 - A leading slash anchors the pattern to the root

duration_ms: 0.246746
...

Subtest: A trailing slash restricts the rule to directories

ok 153 - A trailing slash restricts the rule to directories

duration_ms: 0.212838
...

Subtest: Wildcards respect path separators

ok 154 - Wildcards respect path separators

duration_ms: 0.525541
...

Subtest: Every dot-directory is excluded by .*/

ok 155 - Every dot-directory is excluded by .*/

duration_ms: 0.347759
...

Subtest: Negation re-includes a previously excluded path

ok 156 - Negation re-includes a previously excluded path

duration_ms: 0.423495
...

Subtest: Negation cannot resurrect a file inside an excluded directory

ok 157 - Negation cannot resurrect a file inside an excluded directory

duration_ms: 0.249289
...

Subtest: Last matching rule wins

ok 158 - Last matching rule wins

duration_ms: 0.352321
...

Subtest: Character classes are supported

ok 159 - Character classes are supported

duration_ms: 0.458094
...

Subtest: Paths are normalised before matching

ok 160 - Paths are normalised before matching

duration_ms: 0.338807
...

Subtest: loadIgnoreMatcher merges the three ignore files and skips missing ones

ok 161 - loadIgnoreMatcher merges the three ignore files and skips missing ones

duration_ms: 15.261195
...

Subtest: A repository without ignore files excludes nothing

ok 162 - A repository without ignore files excludes nothing

duration_ms: 1.053256
...

Subtest: The shipped .intentignore excludes build output but keeps sources

ok 163 - The shipped .intentignore excludes build output but keeps sources

duration_ms: 2.963362
...

Subtest: resolveGlobs permits one explicit .intent report without recursively scanning generated runs

ok 164 - resolveGlobs permits one explicit .intent report without recursively scanning generated runs

duration_ms: 9.655215
...

Subtest: Two unrelated AST facts sharing only a file are not linked

ok 165 - Two unrelated AST facts sharing only a file are not linked

duration_ms: 14.620301
...

Subtest: AST facts sharing a symbol are still linked despite the path rule

ok 166 - AST facts sharing a symbol are still linked despite the path rule

duration_ms: 2.20508
...

Subtest: AST details sharing only a file and generic tokens do not create a quadratic subgraph

ok 167 - AST details sharing only a file and generic tokens do not create a quadratic subgraph

duration_ms: 3.129436
...

Subtest: A file-level plan links once to the AST module aggregate instead of every detail

ok 168 - A file-level plan links once to the AST module aggregate instead of every detail

duration_ms: 7.648519
...

Subtest: A shared path still links a plan to an AST fact

ok 169 - A shared path still links a plan to an AST fact

duration_ms: 0.966149
...

Subtest: A bare filename links to a module only when its repository path is unique

ok 170 - A bare filename links to a module only when its repository path is unique

duration_ms: 1.708608
...

Subtest: A bare filename refuses ambiguous module paths

ok 171 - A bare filename refuses ambiguous module paths

duration_ms: 1.068015
...

Subtest: Relations that carry a conclusion survive alongside suppressed noise

ok 172 - Relations that carry a conclusion survive alongside suppressed noise

duration_ms: 2.265753
...

Subtest: Pair ordering stays deterministic across rebuilds

ok 173 - Pair ordering stays deterministic across rebuilds

duration_ms: 2.183748
...

Subtest: Two configuration declarations sharing only a key name are not linked

ok 174 - Two configuration declarations sharing only a key name are not linked

duration_ms: 0.533832
...

Subtest: A shared ticket still connects two configuration declarations

ok 175 - A shared ticket still connects two configuration declarations

duration_ms: 0.454136
...

Subtest: Configuration still links to documentation that describes it

ok 176 - Configuration still links to documentation that describes it

duration_ms: 0.546919
...

Subtest: Configuration file aggregate is the file-level target for an explicit documentation path

ok 177 - Configuration file aggregate is the file-level target for an explicit documentation path

duration_ms: 0.807084
...

Subtest: Configuration aggregates do not create broad capability-topic links

ok 178 - Configuration aggregates do not create broad capability-topic links

duration_ms: 0.511549
...

Subtest: a full six-stage live run passes and reports every stage

ok 179 - a full six-stage live run passes and reports every stage

duration_ms: 3.426661
...

Subtest: a stage that silently fell back to deterministic fails the check

ok 180 - a stage that silently fell back to deterministic fails the check

duration_ms: 0.517167
...

Subtest: a missing stage cannot pass as covered

ok 181 - a missing stage cannot pass as covered

duration_ms: 0.365281
...

Subtest: per-stage and total budgets are enforced separately

ok 182 - per-stage and total budgets are enforced separately

duration_ms: 0.405856
...

Subtest: live request timeout reaches the stage budget without shortening a larger override

ok 183 - live request timeout reaches the stage budget without shortening a larger override

duration_ms: 0.195128
...

Subtest: a stage reason is recorded with provider text redacted

ok 184 - a stage reason is recorded with provider text redacted

duration_ms: 0.80547
...

Subtest: history records the trend without gating on it

ok 185 - history records the trend without gating on it

duration_ms: 0.458017
...

Subtest: recorded audit history includes the current run exactly once

ok 186 - recorded audit history includes the current run exactly once

duration_ms: 0.772135
...

Subtest: history stays chronological, bounded and free of duplicate runs

ok 187 - history stays chronological, bounded and free of duplicate runs

duration_ms: 7.969106
...

Subtest: an audit converts to exactly the redacted fields history keeps

ok 188 - an audit converts to exactly the redacted fields history keeps

duration_ms: 1.174095
...

Subtest: an empty history summarizes without pretending to have measured anything

ok 189 - an empty history summarizes without pretending to have measured anything

duration_ms: 0.201234
...

Subtest: a batched run is measured per record, not per request

ok 190 - a batched run is measured per record, not per request

duration_ms: 6.317728
...

Subtest: a model whose response the validator rejected is not counted as enriched

ok 191 - a model whose response the validator rejected is not counted as enriched

duration_ms: 0.351968
...

Subtest: a failed model is a comparison result rather than a crash

ok 192 - a failed model is a comparison result rather than a crash

duration_ms: 0.89634
...

Subtest: agreement compares only records both models enriched

ok 193 - agreement compares only records both models enriched

duration_ms: 0.364976
...

Subtest: agreement is absent rather than perfect when nothing overlaps

ok 194 - agreement is absent rather than perfect when nothing overlaps

duration_ms: 0.53979
...

Subtest: the rendered comparison names the cheapest and fastest passing model

ok 195 - the rendered comparison names the cheapest and fastest passing model

duration_ms: 0.322858
...

Subtest: semantic LLM modes default to require-llm

ok 196 - semantic LLM modes default to require-llm

duration_ms: 1.889451
...

Subtest: Markdown extractor separates TODO plans and changelog claims

ok 197 - Markdown extractor separates TODO plans and changelog claims

duration_ms: 18.34059
...

Subtest: Markdown extractor preserves indented continuation lines and their source range

ok 198 - Markdown extractor preserves indented continuation lines and their source range

duration_ms: 5.822737
...

Subtest: TODO bare filenames inherit an existing directory from the heading scope

ok 199 - TODO bare filenames inherit an existing directory from the heading scope

duration_ms: 2.23172
...

Subtest: TODO resolves a bare filename only when its repository basename is unique

ok 200 - TODO resolves a bare filename only when its repository basename is unique

duration_ms: 3.861693
...

Subtest: TODO and CHANGELOG resolve the same bare filename to one repository path

ok 201 - TODO and CHANGELOG resolve the same bare filename to one repository path

duration_ms: 2.754075
...

Subtest: CHANGELOG keeps an ambiguous bare filename unresolved

ok 202 - CHANGELOG keeps an ambiguous bare filename unresolved

duration_ms: 2.601177
...

Subtest: Markdown path resolution drops paths and heading scopes outside the repository

ok 203 - Markdown path resolution drops paths and heading scopes outside the repository

duration_ms: 1.189923
...

Subtest: TODO and CHANGELOG receive audited LLM enrichment without changing structural facts

ok 204 - TODO and CHANGELOG receive audited LLM enrichment without changing structural facts

duration_ms: 25.511006
...

Subtest: Markdown enrichment corrects one rejected response and audits both attempts

ok 205 - Markdown enrichment corrects one rejected response and audits both attempts

duration_ms: 4.883962
...

Subtest: large Markdown enrichment uses bounded concurrency and keeps provider audits ordered

ok 206 - large Markdown enrichment uses bounded concurrency and keeps provider audits ordered

duration_ms: 52.81593
...

Subtest: TODO and CHANGELOG LLM fallback and require mode are explicit

ok 207 - TODO and CHANGELOG LLM fallback and require mode are explicit

duration_ms: 2.453443
...

Subtest: TODO and CHANGELOG reject structurally invalid LLM enrichments

ok 208 - TODO and CHANGELOG reject structurally invalid LLM enrichments

duration_ms: 2.612869
...

Subtest: a truncated batch is split and every record keeps its own response provenance

ok 209 - a truncated batch is split and every record keeps its own response provenance

duration_ms: 5.259736
...

Subtest: a malformed batch response splits instead of failing the whole stage

ok 210 - a malformed batch response splits instead of failing the whole stage

duration_ms: 5.600509
...

Subtest: MCP 2026 profile is stateless and exposes discovery plus complete results

ok 211 - MCP 2026 profile is stateless and exposes discovery plus complete results

duration_ms: 2.129088
...

Subtest: MCP 2026 rejects missing metadata and unsupported versions with protocol errors

ok 212 - MCP 2026 rejects missing metadata and unsupported versions with protocol errors

duration_ms: 0.790342
...

Subtest: MCP legacy profile negotiates 2025-11-25 and requires initialize

ok 213 - MCP legacy profile negotiates 2025-11-25 and requires initialize

duration_ms: 0.393414
...

Subtest: MCP exposes annotated intake command/query tools backed by the domain handler

ok 214 - MCP exposes annotated intake command/query tools backed by the domain handler

duration_ms: 12.531923
...

Subtest: An LLM record is marked as inference and keeps runtime-owned provenance

ok 215 - An LLM record is marked as inference and keeps runtime-owned provenance

duration_ms: 35.112278
...

Subtest: NL extraction corrects one rejected structured response and audits both attempts

ok 216 - NL extraction corrects one rejected structured response and audits both attempts

duration_ms: 4.343941
...

Subtest: Confidence must satisfy the provider schema instead of being silently clamped

ok 217 - Confidence must satisfy the provider schema instead of being silently clamped

duration_ms: 8.433762
...

Subtest: Source lines are clamped to the real file

ok 218 - Source lines are clamped to the real file

duration_ms: 2.602579
...

Subtest: A placeholder object is recorded as a missing field, not as content

ok 219 - A placeholder object is recorded as a missing field, not as content

duration_ms: 11.648266
...

Subtest: A real object is kept verbatim and reports no missing field

ok 220 - A real object is kept verbatim and reports no missing field

duration_ms: 2.16964
...

Subtest: The explicit unknown action is reported as a missing field

ok 221 - The explicit unknown action is reported as a missing field

duration_ms: 1.615956
...

Subtest: Both gaps are reported together

ok 222 - Both gaps are reported together

duration_ms: 1.433077
...

Subtest: Out-of-vocabulary enums are rejected instead of changing the provider intent

ok 223 - Out-of-vocabulary enums are rejected instead of changing the provider intent

duration_ms: 4.738106
...

Subtest: Rejected NL output keeps provider metadata in the failed audit

ok 224 - Rejected NL output keeps provider metadata in the failed audit

duration_ms: 2.264881
...

Subtest: The documented confidence hierarchy holds across LLM extractors

ok 225 - The documented confidence hierarchy holds across LLM extractors

duration_ms: 0.902748
...

Subtest: NL extractor produces deterministic non-LLM records

ok 226 - NL extractor produces deterministic non-LLM records

duration_ms: 10.154322
...

Subtest: NL public extraction boundary names a missing sourcePath before path resolution

ok 227 - NL public extraction boundary names a missing sourcePath before path resolution

duration_ms: 0.957022
...

Subtest: deterministic NL fallback skips Markdown headings and recognizes comparison intent

ok 228 - deterministic NL fallback skips Markdown headings and recognizes comparison intent

duration_ms: 4.197273
...

Subtest: path extraction rejects lowercase prose alternations without losing repository paths

ok 229 - path extraction rejects lowercase prose alternations without losing repository paths

duration_ms: 0.873315
...

Subtest: path extraction rejects dotted DSL fields but keeps known file extensions

ok 230 - path extraction rejects dotted DSL fields but keeps known file extensions

duration_ms: 0.263498
...

Subtest: detectModality ignores parenthetical labels and bare adjectives

ok 231 - detectModality ignores parenthetical labels and bare adjectives

duration_ms: 0.552519
...

Subtest: detectModality reads prohibitions and periphrastic obligation as requirements

ok 232 - detectModality reads prohibitions and periphrastic obligation as requirements

duration_ms: 0.562719
...

Subtest: detectPolarity does not treat without-complements as sentence negation

ok 233 - detectPolarity does not treat without-complements as sentence negation

duration_ms: 0.245307
...

Subtest: path extraction rejects HTTP routes, host paths and parent traversal

ok 234 - path extraction rejects HTTP routes, host paths and parent traversal

duration_ms: 0.297079
...

Subtest: symbol extraction rejects hostnames without losing qualified code symbols

ok 235 - symbol extraction rejects hostnames without losing qualified code symbols

duration_ms: 0.373394
...

Subtest: symbol extraction separates repository files and all-caps prose from code identifiers

ok 236 - symbol extraction separates repository files and all-caps prose from code identifiers

duration_ms: 0.210236
...

Subtest: topic keywords normalize paths, camelCase and documentation word forms

ok 237 - topic keywords normalize paths, camelCase and documentation word forms

duration_ms: 0.408341
...

Subtest: NL LLM extraction emits audited provenance and bounded DSL records

ok 238 - NL LLM extraction emits audited provenance and bounded DSL records

duration_ms: 34.182372
...

Subtest: NL LLM failure is explicit when deterministic fallback is used

ok 239 - NL LLM failure is explicit when deterministic fallback is used

duration_ms: 5.018287
...

Subtest: require-llm rejects instead of silently falling back

ok 240 - require-llm rejects instead of silently falling back

duration_ms: 0.679015
...

Subtest: OpenRouter client parses structured JSON without exposing key

ok 241 - OpenRouter client parses structured JSON without exposing key

duration_ms: 26.177383
...

Subtest: OpenRouter client preserves metadata when runtime rejects structured output

ok 242 - OpenRouter client preserves metadata when runtime rejects structured output

duration_ms: 1.736352
...

Subtest: OpenRouter client lists available models after an invalid model ID

ok 243 - OpenRouter client lists available models after an invalid model ID

duration_ms: 10.728128
...

Subtest: OpenRouter JSON timeout is not repeated as a schema fallback request

ok 244 - OpenRouter JSON timeout is not repeated as a schema fallback request

duration_ms: 0.850491
...

Subtest: OpenRouter request obeys a shared pipeline deadline without retrying

ok 245 - OpenRouter request obeys a shared pipeline deadline without retrying

duration_ms: 0.72275
...

Subtest: Documentation extractor converts OpenRouter structured output to bounded LLM records

ok 246 - Documentation extractor converts OpenRouter structured output to bounded LLM records

duration_ms: 21.878429
...

Subtest: Documentation extractor reports and enforces its chunk budget

ok 247 - Documentation extractor reports and enforces its chunk budget

duration_ms: 6.128927
...

Subtest: Documentation extractor corrects one rejected chunk and audits both responses

ok 248 - Documentation extractor corrects one rejected chunk and audits both responses

duration_ms: 5.857127
...

Subtest: Documentation extractor does not spend its correction retry on a timeout

ok 249 - Documentation extractor does not spend its correction retry on a timeout

duration_ms: 3.577088
...

Subtest: Documentation extractor exposes an audited configuration failure

ok 250 - Documentation extractor exposes an audited configuration failure

duration_ms: 0.390609
...

Subtest: Documentation extractor uses bounded concurrent OpenRouter requests

ok 251 - Documentation extractor uses bounded concurrent OpenRouter requests

duration_ms: 38.945505
...

Subtest: LLM summarizer receives graph data and preserves grounded record citations

ok 252 - LLM summarizer receives graph data and preserves grounded record citations

duration_ms: 7.219022
...

Subtest: LLM summarizer validates provider fields before creating semantic IDs

ok 253 - LLM summarizer validates provider fields before creating semantic IDs

duration_ms: 5.645427
...

Subtest: LLM summarizer diagnoses a provider that ignores the response envelope

ok 254 - LLM summarizer diagnoses a provider that ignores the response envelope

duration_ms: 3.062176
...

Subtest: LLM summarizer rejects diagnostic citations outside the supplied graph

ok 255 - LLM summarizer rejects diagnostic citations outside the supplied graph

duration_ms: 4.927535
...

Subtest: LLM summarizer prioritizes documentation over the AST payload budget

ok 256 - LLM summarizer prioritizes documentation over the AST payload budget

duration_ms: 132.334546
...

Subtest: deterministic summary presents AST module aggregates instead of low-level calls

ok 257 - deterministic summary presents AST module aggregates instead of low-level calls

duration_ms: 2.850398
...

Subtest: The summarizer grounds a fabricated record citation from its diagnostic

ok 258 - The summarizer grounds a fabricated record citation from its diagnostic

duration_ms: 4.055261
...

Subtest: The summarizer still fails when the retry fabricates a diagnostic again

ok 259 - The summarizer still fails when the retry fabricates a diagnostic again

duration_ms: 3.300453
...

Subtest: variable contracts and operation plans have deterministic content-bound IDs

ok 260 - variable contracts and operation plans have deterministic content-bound IDs

duration_ms: 5.614727
...

Subtest: every variable grants Founder read/write authority and immutable variables reject other writers

ok 261 - every variable grants Founder read/write authority and immutable variables reject other writers

duration_ms: 0.720905
...

Subtest: plans reject undeclared parameters, actor visibility gaps and payload secrets

ok 262 - plans reject undeclared parameters, actor visibility gaps and payload secrets

duration_ms: 1.363876
...

Subtest: safety-sensitive commands require a Founder decision, a human boundary and verification

ok 263 - safety-sensitive commands require a Founder decision, a human boundary and verification

duration_ms: 1.057251
...

Subtest: plan hash detects semantic tampering

ok 264 - plan hash detects semantic tampering

duration_ms: 1.046037
...

Subtest: compiler emits the exact governed envelope without an execution surface

ok 265 - compiler emits the exact governed envelope without an execution surface

duration_ms: 1.065453
...

Subtest: runtime draft boundaries ignore lifecycle and identity fields injected by untyped callers

ok 266 - runtime draft boundaries ignore lifecycle and identity fields injected by untyped callers

duration_ms: 0.609059
...

Subtest: compiler fails closed on extra, stale, wrong-source and wrong-type bindings

ok 267 - compiler fails closed on extra, stale, wrong-source and wrong-type bindings

duration_ms: 1.471806
...

Subtest: file boundary writes one private envelope atomically and refuses overwrite

ok 268 - file boundary writes one private envelope atomically and refuses overwrite

duration_ms: 13.57971
...

Subtest: Offline pipeline writes a complete run

ok 269 - Offline pipeline writes a complete run

duration_ms: 141.186458
...

Subtest: Pipeline persists synthesis, validation and review patch, then registers approval receipt

ok 270 - Pipeline persists synthesis, validation and review patch, then registers approval receipt

duration_ms: 30.003963
...

Subtest: Pipeline integrates multi-participant communication into graph, diagnostics, reality and run artifacts

ok 271 - Pipeline integrates multi-participant communication into graph, diagnostics, reality and run artifacts

duration_ms: 35.716437
...

Subtest: Pipeline require-llm task synthesis failure is audited and never publishes latest

ok 272 - Pipeline require-llm task synthesis failure is audited and never publishes latest

duration_ms: 12.271371
...

Subtest: Pipeline persists an audited failure when communication require-llm cannot run

ok 273 - Pipeline persists an audited failure when communication require-llm cannot run

duration_ms: 12.266632
...

Subtest: Pipeline persists communication stage failure and does not publish latest

ok 274 - Pipeline persists communication stage failure and does not publish latest

duration_ms: 20.504663
...

Subtest: Pipeline persists a failed manifest when NL require-llm aborts

ok 275 - Pipeline persists a failed manifest when NL require-llm aborts

duration_ms: 1.991047
...

Subtest: Pipeline persists a failed manifest when Markdown require-llm aborts

ok 276 - Pipeline persists a failed manifest when Markdown require-llm aborts

duration_ms: 8.840128
...

Subtest: Pipeline persists a failed manifest for an unexpected summary failure

ok 277 - Pipeline persists a failed manifest for an unexpected summary failure

duration_ms: 10.275784
...

Subtest: Proposal validation reports existing TODO duplicates and orders dependencies before priority

ok 278 - Proposal validation reports existing TODO duplicates and orders dependencies before priority

duration_ms: 17.973545
...

Subtest: Proposal validation rejects dependency cycles and whitespace-only criterion duplicates

ok 279 - Proposal validation rejects dependency cycles and whitespace-only criterion duplicates

duration_ms: 2.345688
...

Subtest: Python package executes the local TypeScript reality runtime without a server

ok 280 - Python package executes the local TypeScript reality runtime without a server

duration_ms: 2119.643435
...

Subtest: runtime cycle extractor turns probes, violations, drift and proposals into records

ok 281 - runtime cycle extractor turns probes, violations, drift and proposals into records

duration_ms: 18.1615
...

Subtest: measurements enter as fact and the cycle own suggestions as inference

ok 282 - measurements enter as fact and the cycle own suggestions as inference

duration_ms: 5.913802
...

Subtest: a violated expectation is negative and blocked, a healthy probe positive and verified

ok 283 - a violated expectation is negative and blocked, a healthy probe positive and verified

duration_ms: 2.619289
...

Subtest: a probe that could not run is unknown, not failing

ok 284 - a probe that could not run is unknown, not failing

duration_ms: 1.520496
...

Subtest: a watched path lets a runtime fact link to the code that produces it

ok 285 - a watched path lets a runtime fact link to the code that produces it

duration_ms: 17.560164
...

Subtest: a document that is not an autonom cycle is refused by schema, not silently empty

ok 286 - a document that is not an autonom cycle is refused by schema, not silently empty

duration_ms: 2.44982
...

Subtest: Runtime validator enforces the complete Intent DSL enum and object contract

ok 287 - Runtime validator enforces the complete Intent DSL enum and object contract

duration_ms: 4.338098
...

Subtest: Linker and remote action boundary reject malformed records before graph construction

ok 288 - Linker and remote action boundary reject malformed records before graph construction

duration_ms: 5.811759
...

Subtest: Graph validator rejects invalid relations and inconsistent statistics

ok 289 - Graph validator rejects invalid relations and inconsistent statistics

duration_ms: 2.519089
...

[t2c:a2a] listening on 127.0.0.1:36077

Subtest: diff UI and TypeScript/Python SDKs use the live backend runtime

ok 290 - diff UI and TypeScript/Python SDKs use the live backend runtime

duration_ms: 234.747876
...

Subtest: MCP/A2A action boundary rejects traversal and symlink escapes

ok 291 - MCP/A2A action boundary rejects traversal and symlink escapes

duration_ms: 24.973522
...

Subtest: bounded retrieval cannot create a relation until a grounded reranker accepts it

ok 292 - bounded retrieval cannot create a relation until a grounded reranker accepts it

duration_ms: 18.962655
...

Subtest: reranker fails closed on ungrounded quotes and more than one accepted module

ok 293 - reranker fails closed on ungrounded quotes and more than one accepted module

duration_ms: 4.074137
...

Subtest: OpenRouter reranking is required, structured and reusable only through an identity-bound cache

ok 294 - OpenRouter reranking is required, structured and reusable only through an identity-bound cache

duration_ms: 74.179342
...

Subtest: published semantic reranker schemas expose the versioned bounded contracts

ok 295 - published semantic reranker schemas expose the versioned bounded contracts

duration_ms: 1.713597
...

Subtest: provider response validation diagnoses the exact property without coercion

ok 296 - provider response validation diagnoses the exact property without coercion

duration_ms: 0.609914
...

Subtest: one structured contract emits the provider schema and parses the same value

ok 297 - one structured contract emits the provider schema and parses the same value

duration_ms: 2.484289
...

Subtest: structured parsing fails closed with the exact response path

ok 298 - structured parsing fails closed with the exact response path

duration_ms: 0.874779
...

Subtest: object uniqueness uses canonical JSON identity rather than property order

ok 299 - object uniqueness uses canonical JSON identity rather than property order

duration_ms: 0.348751
...

Subtest: a short NL symbol resolves to its only AST owner

ok 300 - a short NL symbol resolves to its only AST owner

duration_ms: 20.635731
...

Subtest: an ambiguous short NL symbol does not pretend that either AST owner is selected

ok 301 - an ambiguous short NL symbol does not pretend that either AST owner is selected

duration_ms: 3.748566
...

Subtest: an explicit path selects one owner of an otherwise ambiguous symbol

ok 302 - an explicit path selects one owner of an otherwise ambiguous symbol

duration_ms: 2.217069
...

Subtest: a qualified symbol selects its exact AST declaration without a path

ok 303 - a qualified symbol selects its exact AST declaration without a path

duration_ms: 1.081319
...

Subtest: a symbol and explicit path conflict reports the observed AST location

ok 304 - a symbol and explicit path conflict reports the observed AST location

duration_ms: 0.977542
...

Subtest: missingFields diagnostics prescribe a concrete edit for every known gap

ok 305 - missingFields diagnostics prescribe a concrete edit for every known gap

duration_ms: 0.724937
...

Subtest: Target normalization canonicalizes paths, symbols and cross-language separators

ok 306 - Target normalization canonicalizes paths, symbols and cross-language separators

duration_ms: 2.138032
...

Subtest: Qualified AST symbols align with short plan and documentation targets

ok 307 - Qualified AST symbols align with short plan and documentation targets

duration_ms: 17.368799
...

Subtest: Structured task synthesis materializes stable, grounded contracts with a complete audit

ok 308 - Structured task synthesis materializes stable, grounded contracts with a complete audit

duration_ms: 45.99265
...

Subtest: blank response-local proposal keys are rejected instead of invented by the runtime

ok 309 - blank response-local proposal keys are rejected instead of invented by the runtime

duration_ms: 6.79877
...

Subtest: prefer-llm exposes raw diagnostic actions without claiming semantic task generation

ok 310 - prefer-llm exposes raw diagnostic actions without claiming semantic task generation

duration_ms: 1.267921
...

Subtest: communication divergence is grounded in task synthesis without treating agent claims as facts

ok 311 - communication divergence is grounded in task synthesis without treating agent claims as facts

duration_ms: 5.209256
...

Subtest: task synthesis defaults to require-llm and fails explicitly without a provider

ok 312 - task synthesis defaults to require-llm and fails explicitly without a provider

duration_ms: 1.273814
...

Subtest: invalid structured LLM citations are rejected or visibly degraded according to mode

ok 313 - invalid structured LLM citations are rejected or visibly degraded according to mode

duration_ms: 10.452399
...

Subtest: task synthesis timeout is audited and never retried as a format fallback

ok 314 - task synthesis timeout is audited and never retried as a format fallback

duration_ms: 16.464353
...

Subtest: A fabricated record citation is grounded from its cited diagnostic without a retry

ok 315 - A fabricated record citation is grounded from its cited diagnostic without a retry

duration_ms: 4.844261
...

Subtest: A fabricated diagnostic still fails after the corrective retry

ok 316 - A fabricated diagnostic still fails after the corrective retry

duration_ms: 5.583099
...

Subtest: TensorFlow remains an explicit fallback when the isolated adapter is not installed

ok 317 - TensorFlow remains an explicit fallback when the isolated adapter is not installed

duration_ms: 7.394806
...

Subtest: TODO patch rendering is stable, dependency-first and excludes classified duplicates

ok 318 - TODO patch rendering is stable, dependency-first and excludes classified duplicates

duration_ms: 19.761122
...

Subtest: empty and duplicate-only results render an explicit no-op patch

ok 319 - empty and duplicate-only results render an explicit no-op patch

duration_ms: 3.543215
...

Subtest: apply rejects missing or wrong approval, stale TODO and a tampered patch

ok 320 - apply rejects missing or wrong approval, stale TODO and a tampered patch

duration_ms: 16.394695
...

Subtest: approved apply is atomic, receipt-backed and idempotent

ok 321 - approved apply is atomic, receipt-backed and idempotent

duration_ms: 9.500674
...

Subtest: service actions execute LLM propose -> render -> approved apply with scoped artifacts

ok 322 - service actions execute LLM propose -> render -> approved apply with scoped artifacts

duration_ms: 38.464243
...

Subtest: scanTree prunes ignored directories and records file signatures

ok 323 - scanTree prunes ignored directories and records file signatures

duration_ms: 16.387029
...

Subtest: diffSnapshots classifies additions, modifications and removals

ok 324 - diffSnapshots classifies additions, modifications and removals

duration_ms: 0.443609
...

Subtest: describeDelta truncates long change lists

ok 325 - describeDelta truncates long change lists

duration_ms: 0.200925
...

Subtest: An unchanged tree produces exactly one report and then stays quiet

ok 326 - An unchanged tree produces exactly one report and then stays quiet

duration_ms: 3.76991
...

Subtest: Reports are rate limited to one per interval no matter how often files change

ok 327 - Reports are rate limited to one per interval no matter how often files change

duration_ms: 26.688159
...

Subtest: A change is reported once the interval has elapsed

ok 328 - A change is reported once the interval has elapsed

duration_ms: 2.817886
...

Subtest: Ignored files never trigger a report

ok 329 - Ignored files never trigger a report

duration_ms: 4.678022
...

Subtest: A failing report is surfaced and does not stop the watcher

ok 330 - A failing report is surfaced and does not stop the watcher

duration_ms: 1.51215
...

Subtest: --no-initial-report waits for a real change

ok 331 - --no-initial-report waits for a real change

duration_ms: 1.251563
...

Subtest: Communication changes trigger watch and coalesce under the existing report rate limit

ok 332 - Communication changes trigger watch and coalesce under the existing report rate limit

duration_ms: 3.311932
...

Subtest: workflow verifier rejects duplicate top-level YAML keys

ok 333 - workflow verifier rejects duplicate top-level YAML keys

duration_ms: 77.272234
...

Subtest: workspace headline trend ignores AST-only topic and source churn

ok 334 - workspace headline trend ignores AST-only topic and source churn

duration_ms: 1.456164
...

Subtest: workspace comparison measures origin/main against uncommitted filesystem intent

ok 335 - workspace comparison measures origin/main against uncommitted filesystem intent

duration_ms: 326.429539
...
1..335

tests 335

suites 0

pass 334

fail 0

cancelled 0

skipped 1

todo 0

duration_ms 10600.724353: 335 tests, 334 pass, 0 fail, 1 local JDK skip

  • fresh Docker : pass
  • Docker : independently owned stale , expected exit 101 at

Follow-up failure-path validation

  • actual LiteLLM HTTP stack + local 404 endpoint: NotFoundError in ~705 ms, exactly one request, max_tokens=8192
  • slow local endpoint + 0.5 s request limit: Timeout in ~799 ms, exactly one request
  • fresh npm run verify: 335 tests, 334 pass, 0 fail, 1 local JDK skip
  • fresh Docker e2e-core: pass
  • Docker e2e-full: independently owned stale sdk/rust/Cargo.lock, expected exit 101 at cargo fetch --locked

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant